Towards a systematic approach to manual annotation of code smells

Code smells are structures in code that may indicate maintainability issues. They challenging to define, and software engineers detect them differently. Mitigation of this problem could be an AI smell detector. However, develop it, we need a standardized benchmark dataset. Existing datasets suffer from (1) annotation subjectivity, (2) lack ground-truth consensus among annotators, (3) reproducibility This paper aims systematic manual procedure addresses these We tailored the prescriptive natural language processing methodology detection: cross-validate annotations mitigate clear guidelines reach consensus, follow literature recommendations for open-source our tools extracted existing empirical research. The annotators refined their understanding task through proof-of-concept encompassing retrospective discussion disagreement resolution then performed full annotation. confirmed was reached by measuring consistency. Our contributions proposed procedure, novel dataset C# projects, annotators' experience report, open-sourced supporting tool.